Communication Patterns

Order service publishes "order.created" event
                            ↓
        Multiple subscribers listen:
        - Email service: sends confirmation
        - Inventory service: decrements stock
        - Analytics service: logs metrics
        - Notification service: sends push notification
        
Each subscriber processes independently

Pending tasks → Message Queue
              ↓
         Worker 1: process
         Worker 2: process
         Worker 3: process
         
Each task processed by exactly one worker
If worker fails, queue re-delivers to another worker

User clicks "Pay" → 
  1. Sync: Validate card (must be instant)
         Process payment (stripe API call)
         If success → return confirmation to user
         If fail → return error immediately
  
  2. Async: After sync success
         - Update order status
         - Send confirmation email
         - Log audit trail
         - Update analytics
         - Send receipt SMS

Payment request (sync) → Stripe API → DB update
                         ↓
                    Success/Failure
                         ↓
                    If success: queue async tasks
                         ↓
            [Email, SMS, Analytics, Audit log]
            (process in background)

User tweets → Tweet service publishes "tweet.created"
                            ↓
Multiple subscribers:
  1. Notification service → Push notifications to followers
  2. Timeline service → Update follower timelines
  3. Search service → Index tweet for search
  4. Analytics service → Log tweet metrics
  5. Cache service → Update Redis caches

Each subscriber processes independently
If notification service crashes, tweet still indexed and cached

Influencer tweets → 50M followers
1 event → 50M notifications needed

With Pub/Sub:
- Publish once
- Notification service scales horizontally (1000 workers)
- Each worker handles 50K notifications

With Queue:
- Would need 50M messages in queue
- Inefficient

Normal load (100 req/sec):
  Request → Process (sync)
         ↓
      DB update
         ↓
      Return response (50ms)

Traffic spike (10,000 req/sec):
  Request → Queue (instant)
         ↓
      Return "accepted" (1ms)
         ↓
    Workers process from queue at 500 req/sec
         ↓
    Takes ~20 seconds to clear spike

Without queue:
  10,000 requests hit service
  Service crashes (can't handle)
  Users get 500 errors

@app.post("/process")
def process_job(data):
    # Instead of processing here:
    # db.process(data)  # Would crash under load
    
    # Queue it:
    queue.push("jobs", json.dumps(data))
    return {"status": "queued", "job_id": uuid()}

# Separate worker pool
def worker():
    while True:
        job = queue.pop("jobs")
        db.process(json.loads(job))
        # Can scale workers independently

WebSocket (1 million concurrent):
- Each connection = TCP socket + memory state
- Sticky session required (user always routes to same server)
- 10 servers × 100K connections = complex state mgmt
- Need WebSocket-aware load balancer
- Memory overhead: ~1KB per connection = 1GB for 1M

Long Polling (1 million concurrent):
- Each active poll = HTTP request
- Stateless (can go to any server)
- Load balancer distributes freely
- Memory overhead minimal
- More HTTP requests (higher CPU)
- Better for CDN/simple infrastructure

- WebSocket for active users (actively using app)
- Long Polling fallback for inactive (load reduction)
- Or: SSE as middle ground (stateless, server-push only)

Task fails
  ↓
Retry attempt 1 (after 1 second)
  ↓
If still fails:
Retry attempt 2 (after 2 seconds)
  ↓
If still fails:
Retry attempt 3 (after 4 seconds)
  ↓
If still fails (max retries):
Move to DLQ (manual investigation)

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(5),  # Max 5 attempts
    wait=wait_exponential(
        multiplier=1,  # 1, 2, 4, 8, 16 seconds
        min=1,
        max=60
    )
)
def process_payment(order_id):
    try:
        stripe.charge(order_id)
    except TemporaryError as e:
        # Transient error, retry
        raise
    except PermanentError as e:
        # Don't retry, send to DLQ
        dlq.push("failed_payments", order_id)
        return

def dlq_processor():
    # Manually inspect failed tasks
    for order_id in dlq.get_all("failed_payments"):
        admin_alert(f"Payment failed for order {order_id}")